Skip to content

[SDK] test: Add e2e test for tune function. #2399

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 23 commits into from
Aug 6, 2024
Merged

[SDK] test: Add e2e test for tune function. #2399

merged 23 commits into from
Aug 6, 2024

Conversation

@Electronic-Waste
Copy link
Member

@Electronic-Waste Electronic-Waste commented Jul 30, 2024

What this PR does / why we need it:

  • Adds an e2e test in the run_e2e_experiment.py for the tune function.
  • Change the error field name from metrics_collector to metrics_collector_spec

Which issue(s) this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close the issue(s) when PR gets merged):
Fixes #2383

Checklist:

  • Docs included if any changes are user facing

@Electronic-Waste
Copy link
Member Author

Electronic-Waste commented Jul 30, 2024

PTAL👀 @andreyvelich @johnugeorge @tenzen-y @tariq-hasan

@Electronic-Waste Electronic-Waste mentioned this pull request Jul 30, 2024
Closed
6 tasks
andreyvelich
andreyvelich reviewed Jul 30, 2024
View reviewed changes
Copy link
Member

@andreyvelich andreyvelich left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks a lot for doing this @Electronic-Waste!
This should help us to make this API more stable.

@google-oss-prow google-oss-prow bot added size/L and removed size/M labels Jul 30, 2024
@Electronic-Waste Electronic-Waste force-pushed the fix/tune branch 3 times, most recently from 6235a39 to b371915 Compare July 30, 2024 18:10
@Electronic-Waste
Copy link
Member Author

Electronic-Waste commented Aug 1, 2024

Now run-e2e-tune-api.py has been added, and the error in CI/CD test is addressed.

Also, I found function verify_experiment_results is used in both the tune API test and the original test. So I move this function to verify.py and import it separately in run-e2e-tune-api.py and run-e2e-experiment.py.

Could you take a look?👀 @andreyvelich

andreyvelich
andreyvelich reviewed Aug 1, 2024
View reviewed changes
@Electronic-Waste
Copy link
Member Author

Electronic-Waste commented Aug 1, 2024

@andreyvelich I've added a new workflow file e2e-test-tune-api.yaml to .github/workflows, and also, e2e-test-tune-api.sh to test/e2e/v1beta1/scripts/gh-actions. Now, the test for tune API is separate from other tests.

andreyvelich
andreyvelich reviewed Aug 1, 2024
View reviewed changes
Copy link
Member

@andreyvelich andreyvelich left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Overall looks good, thank you @tenzen-y, just a few comments.
/assign @johnugeorge @tenzen-y

@Electronic-Waste
Copy link
Member Author

Electronic-Waste commented Aug 1, 2024

This PR is also related to my GSoC project.

ref issue: #2340

tenzen-y
tenzen-y reviewed Aug 1, 2024
View reviewed changes
@Electronic-Waste
Copy link
Member Author

Electronic-Waste commented Aug 2, 2024

I've removed the time.sleep and refactored the workflow file with template.

PTAL👀 @tenzen-y if you are available.

andreyvelich
andreyvelich reviewed Aug 5, 2024
View reviewed changes
.github/workflows/e2e-test-tune-api.yaml
experiments: ""
# Comma Delimited
trial-images: ""
tune-api: true
Copy link
Member

@andreyvelich andreyvelich Aug 5, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@tenzen-y Does it look to you on how we separate the template-e2e-test ?

@Electronic-Waste
Copy link
Member Author

Electronic-Waste commented Aug 6, 2024

@andreyvelich I've modified this PR as you suggested in the last few comments. Could you plz take a look when you are available? Thanks so much!

andreyvelich
andreyvelich approved these changes Aug 6, 2024
View reviewed changes
Copy link
Member

@andreyvelich andreyvelich left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks a lot for doing this @Electronic-Waste!
It should improve tune API stability.
/lgtm
/approve

@google-oss-prow google-oss-prow bot added the lgtm label Aug 6, 2024
@google-oss-prow
Copy link

google-oss-prow bot commented Aug 6, 2024

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: andreyvelich

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@google-oss-prow google-oss-prow bot merged commit b6f7cfd into kubeflow:master Aug 6, 2024
@andreyvelich andreyvelich mentioned this pull request Nov 25, 2024
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@andreyvelich andreyvelich andreyvelich approved these changes

@anencore94 anencore94 Awaiting requested review from anencore94

@gaocegege gaocegege Awaiting requested review from gaocegege

@johnugeorge johnugeorge Awaiting requested review from johnugeorge

@tenzen-y tenzen-y Awaiting requested review from tenzen-y

Assignees

@johnugeorge johnugeorge

@andreyvelich andreyvelich

@tenzen-y tenzen-y

Labels

approved lgtm size/L

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Add e2e test for tune in the katib_client module

4 participants

@Electronic-Waste @andreyvelich @tenzen-y @johnugeorge